Multi-band YCbCr noise modeling and noise reduction based on scene metadata

ABSTRACT

Image enhancement is achieved by separating image signals, e.g. YCbCr image signals, into a series of frequency bands and performing noise reduction on bands below a given frequency but not on bands above that frequency. The bands are summed to develop the image enhanced signals. The YCbCr, multi-band approach to denoising is able to operate independently—and in an optimized fashion—on both luma and chroma channels. Noise reduction is done based on models developed for both luma and chroma channels by measurements taken for multiple frequency bands, in multiple patches on the ColorChecker chart, and at multiple gain levels, in order to develop a simple yet robust set of models that may be tuned off-line a single time for each camera and then applied to images taken by such cameras in real-time without excessive processing requirements and with satisfactory results across illuminant types and lighting conditions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to commonly-assigned U.S. patent applicationSer. No. 13/492,302 entitled “Method of and Apparatus for ImageEnhancement,” filed Jun. 8, 2012, which is hereby incorporated byreference in its entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

The invention is related to image processing of captured images. Moreparticularly, it is related to multi-band denoising of images.

Description of the Related Art

The image signal processor (ISP) takes the raw image from the imagesensor, and then optionally performs one or more of several operations,such as: gain, binning (in low-light), noise reduction, local tonemapping, demosaicing, white balancing, gamma, filtering, and colorenhancement. The ISP provides a YCbCr (i.e., luma-chroma) image, whichis later compressed. In most cases, post-ISP operations are performed inYCbCr space, not in RGB (i.e., red-green-blue) space. There are severalchallenges associated with obtaining an accurate YCbCr noise model.First, the noise characteristics of images obtained by a digital cameraare quite complicated. They may have signal dependence, e.g., due toshot noise in the image sensor and gamma operations in the camerapipeline, frequency dependence caused by demosaicing, luma sharpening,chroma band-limiting, and binning, inter-channel correlation resultingfrom demosaicing and color correction, and channel dependence caused bywhite balancing. In short, the noise may have signal-wise, channel-wise,and band-wise dependencies.

The noise reduction pipeline in a typical consumer digital camera, e.g.,a mobile phone camera, is fairly basic. It is constrained by the numberof delay lines available for the image signal processor, as well ascomputational limitations. Second, since it typically takes a few yearsto design, test, and produce an image signal processor; the noisereduction algorithm is typically a few generations old by the time adevice makes it to market. The camera pipeline may introduce a number ofartifacts, such as false edges, sprinkles, and black/white pixel clumpsthat, from a signal point-of-view, are not noise, but actually appearmore like structure. These artifacts severely degrade image quality inbright light, especially in the sky regions (i.e., “blue-sky noise”),but they are especially severe in low-light. One way to mitigate noiseas well as artifacts is to increase exposure time so that more photonscan be accumulated in the sensor, but this introduces motion blur.Another way to mitigate noise is to use a sensor that has larger wellcapacity, such as four-thirds, APS-C, or full frame. These sensors areused in DSLR cameras and are quite expensive. Also, they are physicallylarger, requiring more space and making them infeasible for the thinform factors of most modern-day mobile phone cameras.

An accurate noise estimate is important when a measure of localsimilarity is desired. For instance, in denoising, pixels that aresimilar in value to the pixel currently being denoised are typicallyaveraged together in some fashion. Performance in such denoisingoperations is directly dependent on the quality of the “similar pixel”set, which in turn is dependent on the “similarity measure.” Robustsimilarity measures may, therefore, preferably rely on an effective andaccurate noise model in order to be able to adapt to imaging conditions.If the noise model is accurate, the similarity measure can help todifferentiate between signal and noise. Similarly, in applications wheremultiple frames are fused (e.g., high dynamic range imaging) orstitching is involved (e.g., panoramic imaging), there may be a need todifferentiate between still and moving objects within the images, aswell as compensate for registration errors. Relative motion betweenframes can result from three main factors: 1) object motion; 2) camerashake; and 3) rolling shutter. An accurate noise model, such as thatdescribed herein, may help to differentiate between signal and motionfor various imaging conditions, moving objects, and hand shake. If thesimilarity measure is based on an accurate noise model, it will be ableto adapt to changing conditions, object motion, and jitter—resulting infewer “ghosting” artifacts in the image and better overall image qualityin a wide variety of imaging conditions.

SUMMARY OF THE INVENTION

Embodiments according to the present invention provide image enhancementby separating the image signals, e.g., a YCbCr image signal, into aseries of frequency bands and performing noise reduction independentlyon each of the bands using a noise model that is based on cameraproperties and scene metadata. The noise model for a particular cameramay be constructed “offline” (i.e., in the laboratory or at amanufacturing facility) once for the particular camera, and may takeinto account various types of camera metadata, such as: analog anddigital camera gains, signal-to-noise ratios, lux levels, special modes(e.g., binning), white balance gains, etc. In one embodiment, an affinelinear model may be used for the luma channel, wherein the noise modeldepends on signal, analog and digital camera gains, a non-linear scalingfunction for luma that scales the full-band noise to each sub-band, anda ‘system-wide’ gamma estimate for the luma channel. According to someembodiments, a chroma noise model may also be applied, wherein the noisemodel depends on, e.g., camera gains, white balance gains, a non-linearscaling function for chroma that scales the full-band noise to eachsub-band and a ‘system-wide’ gamma estimate for the chroma channel. Suchnoise models may be used to effectively characterize band-wise,signal-wise, and channel-wise YCbCr dependencies in the image signal.The end result is a computationally efficient, yet powerful, way tocharacterize YCbCr noise that can be used in a number of on-device—aswell as off-device—applications across a broad array of camera devices.

In some embodiments, noise reduction is performed on bands below a givenfrequency but not on bands above that frequency. The bands are summed todevelop the image enhanced signals. This results in improved sharpnessand masking of image processing pipeline artifacts. In otherembodiments, each band may be filtered independently once the noisemodel has been applied to the band.

In still other embodiments, the noise model may be spatially-dependent,e.g., taking into account lens shading gains, local tone mapping (LTM)operations, and/or fusion strength, so that it may spatially adapt tochanging noise levels across different parts of the image.

BRIEF DESCRIPTION OF THE FIGURES

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate an implementation of apparatusand methods consistent with the present invention and, together with thedetailed description, serve to explain advantages and principlesconsistent with the invention.

FIG. 1 is a block diagram of a device, according to one embodiment.

FIG. 2 is an exemplary camera-processing pipeline, according to oneembodiment.

FIG. 3 is a block diagram of a multi-band decomposition filter chain,according to one embodiment.

FIG. 4 is a group of graphs showing measured signal-to-noise levels forvarious frequency bands, according to one embodiment.

FIG. 5 is a block diagram of a multi-band denoising scheme, according toone embodiment.

FIG. 6 is a block diagram of a multi-band locally-adaptive denoisingscheme, according to one embodiment.

FIG. 7 is a block diagram of an exemplary image processing device,according to one embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 is a block diagram of an exemplary device 100, such as a cameraor phone. An imager 102, as typical in such devices, is connected to animage processor 104. The image processor 104 is connected to storage 106for both processing storage and longer-term storage after completion ofprocessing. The image processor 104 is also connected to a generalprocessor 108, which performs more general duties. The general processor108 is connected to a display 110 for providing a user the ability toview the current or previously stored images, which the generalprocessor 108 retrieves from storage 106. Storage 106 also stores thefirmware and other software used by the image processor 104 and generalprocessor 106 that perform the preferred embodiments. This is a verygeneral overview, and many variations can be developed, such ascombining the image processor and general processor or forming the imageprocessor using hardware, FPGAs, or programmed DSPs or some combination,as known to those skilled in the art.

FIG. 2 shows a block diagram of an exemplary camera pipeline 200 thatreceives the output of the imager 102. The imager 102 sends a signalthat has Gaussian, white and uncorrelated noise but has signal leveldependence. The noise also may have missing pixels (e.g., if it has beenmosaiced). A gain component 202 produces high gain in low-lightconditions and low gain at high-light conditions. Signal-noise behaviorchanges accordingly. A white balance component 204 changes gains for R,G, and B channels depending on illumination and illuminant type.Furthermore, channel dependence exists after the white balance component204. After using a demosaicing component 206 to demosaic complete RGBplanes, there is frequency dependent inter-channel correlation.Specifically, G-channel high frequency noise is copied to the B and Rchannels, maintaining higher inter-channel correlation than at lowfrequency. After a color correction matrix component 208, theinter-channel correlation is more complicated. After a gamma component210, strong level dependence is added, and the noise may no longer beassumed to be Gaussian. An RGB to YCbCr conversion 212 may be used toconvert to luma and chroma signals, adding an additional inter-channeldependence, as well as channel-wise, signal-wise, and band-wise noise.In particular, in the RAW domain, the same scaling is applied to allchannels, but luma and chroma values are a result of differentoperations in the pipeline (e.g., luma is typically sharpened, whilechroma is typically band-limited). Also, white balancing and colorcorrection operations performed in the pipeline tend to have a biggerimpact on chroma data. Thus, a constant scaling factor in the RAW domainwill translate to different scaling factors for luma and chromachannels. A sharpening/filtering component 214 may boost the lumasignal's high frequency components and bandlimit the chroma signals,causing additional frequency dependence. Denoising according to variousembodiments of the present invention may be applied by denoiser 216.Additional compression may be done in the compress unit 218.

Rather than view this problem from the prism of image denoising,embodiments according to the present invention treat it from theperspective of image enhancement. The goal is to preserve a sharpimpression, avoid a plastic-like look, and remove objectionable low- andmid-frequency noise, as well as retain a certain amount of morevisually-preferable, high-frequency noise (which has the added benefitof masking image processing pipeline artifacts, as well). All theseeffects, in general, result in a more pleasing look to the resultingdenoised image.

Turning now to FIG. 3, an exemplary block diagram of a multi-banddecomposition filter chain 300 is shown. As will be discussed in furtherdetail below, decomposing an image signal into multiple bands isessential for accurately modeling the non-white nature of YCbCr noise.What this means is that, while the full-band YCbCr noise is not white,it is reasonable to assume it to be band-wise white. The number of bandsdepends on pipeline characteristics and desired accuracy. The incomingluma data (Y) (or chroma data, in the case of a chroma decompositionfilter chain) may first be provided to a first low pass filter (LPF)302. The incoming luma data has the output of the first low pass filter302 subtracted by subtracting junction 304. The output of thesubtracting junction 304, i.e., the highest frequency band data, isreferred to herein as Y_(H). The output of the first low pass filter 302is provided to downsampler 306. In some embodiments, the image data maybe down-sample by a factor of two in each direction. Thus, an inputimage that is 8 MP is size will be 2 MP in size after being downsampledonce, 0.5 MP after being downsampled twice, 125 KP after beingdownsample a third time, and so forth. The output from downsampler 306may be provided to a second low pass filter 310. The second low passfilter 310 has a bandwidth similar to that of the first low pass filter302, as preferably all of the bands are equal, though different sizebands could be used if desired. The output of the second low pass filter310 is provided to the subtraction input of the subtracting junction308, so that the output of the subtracting junction is the luma data inthe second-highest frequency band, referred to herein as Y_(LH). Thischain continues until the final low pass filter 316 and the finalsubtracting junction 314. These both receive the luma data that has allbut the two highest frequency bands removed. The final low pass filter316 removes the third highest frequency band and provides its output tothe final downsampler 318, so that the output is the luma data with thethree highest frequency bands removed, referred to herein as Y_(LH). Inthis manner, the multiple bands are separated using the banddecomposition filter bank. Note that, similar decomposition may beachieved using high pass filters (HPFs) instead of LPFs. Also note that,in this embodiment, we have decomposed the signal into four bands.Depending on the desired quality and computational budget, less or morebands may be used.

Turning now to FIG. 4, a group of graphs (400/410/420/430) showingmeasured signal-to-noise levels for various frequency bands areillustrated, according to one embodiment. More particularly, theexemplary graphs shown in FIG. 4 depict sub-band noise for neutral path5 in the ColorChecker chart. Across the x-axis of the graphs is the“image #” field, representing a series of 21 images of neutral patch 5under an indoor ‘A’ illuminant, taken at increasing gain levels, fromleft to right. As will be understood, similar information may becaptured for each of the color patches on the ColorChecker chart andincorporated into the noise model. The y-axes of the graphs representnoise in the particular sub-band being plotted. As may now be understoodwith reference to the exemplary multi-band decomposition filter chainblock diagram described in FIG. 3, graph 400 corresponds to band H(i.e., the highest frequency bands), graph 410 corresponds to band LH,graph 420 corresponds to band LLH, and graph 430 corresponds to bandLLLH (i.e., the lowest frequency bands). In each graph, the ‘a’ linerepresents noise in the luma channel, σ_(Y); the ‘b’ line representsnoise in the blue-difference chroma channel, σ_(Cb); and the ‘c’ linerepresents noise in the red-difference chroma channel, σ_(Cr).

As may now be more fully appreciated, noise typically increases withgain until gain maxes out (around image #17), and then, as the imagebecomes darker, it decreases. Further, in low frequency sub-bands (e.g.,band LLH and band LLLH), chroma noise dominates luma noise. Theseinsights and others described herein were used to help construct themulti-band YCbCr noise model based on scene metadata that will now bedescribed in greater detail.

A Multi-Band YCbCr Noise Model Based on Scene Metadata

Prior art approaches performing noise estimation on RGB image datarather than YCbCr data cannot exploit Human Visual System (HVS)properties that correspond to specifically to luma/chroma space. Forexample, the HVS is more sensitive to changes in brightness than it ischanges in color, and it is less sensitive to high-frequency noise thanit is to low-frequency noise. Further, such prior art approaches eithermake restrictive assumptions that are not reasonable—or that are toocomplex to run on consumer-level camera devices in real-time. Examplesof restrictive assumptions include assumptions regarding noise variancepropagation from RAW and assumptions regarding gamma's continuity. Forexample, rather than implement gamma as a continuous non-linearfunction, often times it is implemented as a piecewise-linear function,breaking down into several distinct, linear regions, e.g.: shadows, twomid-tones, and highlights. There are two main advantages to thisapproach: 1) linear functions are easier to implement; and 2) selectiveregions may be boosted or attenuated independently. However, one of themain disadvantages of this approach is that the resulting function isnot continuous at the “transition points” between the distinct regions.Hence, noise propagation techniques that rely on first order derivativeswill have a sharp spike at these non-continuous transition points.

Since the noise modeling functions used in prior art approaches are notbased on an actual noise model, they don't adapt very well to changinglight and brightness levels. Moreover, to the inventors' knowledge, achroma noise model has not been addressed by the prior art approaches.Finally, none of these prior art approaches effectively utilize camerametadata, e.g., analog and digital camera gains, signal-to-noise ratios,lux levels, special modes (binning on/off), white balance gains, etc.,if the formation of a noise mode. Thus, it would be desirable to have amulti-band, YCbCr noise model based on scene metadata that effectivelycharacterizes band-wise, signal-wise, and channel-wise YCbCrdependencies.

The inventors' process of deriving a YCbCr noise model for a particularcamera involved taking images of a standard, 24-patch ColorChecker ColorRendition Chart under several illuminants for varying light levels usinga studio light. The RGB images were then converted to YCbCr. Then, asimple low-pass filter was used to split the Y, Cb, and Cr channels intosub-bands. At every band-split, the image was sub-sampled by two in eachdimension. The number of sub-bands needed to fully characterize thenoise model for a given implementation may depend on sensorcharacteristics. In some embodiments, four sub-bands may be used. Next,the color patches were extracted from each image, and the mean andstandard deviation of each patch were computed in each band for each ofthe Y, Cb, and Cr channels. The mean serves as the signal value, whilethe standard deviation serves the corresponding noise amount within ineach band. Additionally, camera parameters were captured for each shot,such as: analog gain, digital gain, lux level, signal-to-noise ratio,white balancing gains, color correction matrix, binning mode on/off,etc.

Once a set of signal-to-noise tables has been acquired for differentcamera parameters, the dependence of noise on camera parameters can bestudied so as to isolate important metadata. Finally, an analyticalmodel may be fit to the data that relates noise amount to input signaland various camera parameters.

According to one embodiment of the derived multi-band YCbCr noise model,the full-band noise is not considered to be “white” (i.e., independentof frequency), but the sub-band noise is approximated to be white withineach individual sub-band. The number of bands used in any particularmodel depends on how many bands are required to approximate the noise asband-wise white. According to one embodiment, an affine linear model isused for modeling noise in the luma channel, wherein the noise modeldepends on signal, analog and digital camera gains, a non-linear scalingfunction for luma that scales the full-band noise to each sub-band, anda system-wide gamma for luma. The non-linear scaling function maycomprise a band number- and gain-dependent non-linear function that isused to relate full-band noise to sub-band noise. The advantage of usingsuch a function is that it provides a single expression for the noisemodel that may be used to predict noise for each of the sub-bands. Ifsuch a function is not used, then a different expression may be requiredfor each sub-band. According to another embodiment, a chroma noise modelis based on camera gains, white balance gains, a non-linear scalingfunction for chroma that scales the full-band noise to each sub-band,and a system-wide gamma for chroma.

Thus, according to one embodiment, the inventors have determined a lumanoise model as follows:

First, a bandScale value may be computed according to the followingequation that is both gain- and band-dependent, and which may be usedfor non-linear scaling of the data:bandScale=2^(1−band)/power(gain,2^(1−band)/bandFactorY);where band is the band number (e.g., 1, 2, 3, 4), gain is the product ofAGC (i.e., analog gain), ispDGain (i.e., the ISP's digital gain), andsensorDGain (i.e., the image sensor's digital gain).Next, a maximum noise level, maxSigma, may be defined as follows:maxSigma=power(gain,gammaY)*bandScale;Next, a bias value for the linear model may be defined as follows:bias=maxSigma/biasFactor;(where biasFactor is the minimum noise standard deviation, i.e., atsignal level 0).The slope of the linear modeling function (assuming, in this case, alinear model is being used) is then defined as follows:

if gain <= 8 (i.e., the max analog gain) slope = maxSigma/slopeFactor;else slope = maxSigma/(2*slopeFactor); (where slopeFactor is the rate atwhich noise standard deviation increases with signal).Finally, the luma noise value is calculated using a simple“y=ax+b”-style linear equation model, as follows:ySigma=min((bias+signal*slope),maxSigma).

According to another embodiment, the inventors have determined a chromanoise model as follows:

First, a bandScale value may be computed according to the followingequation that is band-dependent, and which may be used for non-linearscaling of the data:bandScale=power(bandFactorC,−band/1.9)*power(band,2.0);where bandFactorC is a value estimated from the data that scales noisebetween each band in the respective chroma channel.The chroma noise values may then be calculated using the following powerequations:cbSigma=power(AWBBGain*gain,gammaC)*bandScale;crSigma=power(AWBRGain*gain,gammaC)*bandScale;where AWBBGain and AWBRGain are the blue-channel and red-channel whitebalance gains, respectively, and gammaC is the overall gamma of therespective chroma channel. The gamma is described as an “overall” gammahere because several factors may contribute to level dependence, e.g.,base gamma and local tone mapping.

According to other embodiments, additional metadata may be incorporatedinto the model, such as signal-to-noise ratio, lux levels, special modes(e.g., binning on/off), etc. According to still other embodiments, itmay be desirable to have a model that provides more accurate fit to themeasured signal and noise data (i.e., the data gathered from the imagesof the ColorChecker Color Rendition Chart) within each band than alinear model, e.g., a polynomial model, such as a quadratic or cubicmodel. Yet another embodiment may involve devising a noise model for thecamera pipeline before gamma is applied and using “unscentedtransformation” (UT) to accurately predict noise after gamma is applied.Unscented transformation has been developed for control systems topredict noise mean and standard deviations after non-linear operations.As mentioned above, noise propagation methods often rely on first orderderivatives to predict noise levels after gamma is applied. However,even if the gamma function is continuous, this method may not work verywell in low light conditions. In such situations, predicting noise meanand standard deviations via unscented transform is quite robust.

YCbCr Multi-Band Noise Reduction

Nearly all noise reduction algorithms use some sort of ‘similarityscore’ to determine if the pixels neighboring the pixel that is beingdenoised are close enough in value to be included in the denoisingcalculation. According to some embodiments, the value of the pixel beingdenoised is simply replaced by the average of all ‘similar’ pixels inthe neighborhood. However, there are many ways in which a particularimplementation can define what a ‘similar’ pixel is. According to oneembodiment, the following equation may be used to define a ‘similar’pixel:|y _(c) −y _(i) |<Xσ,where y_(c) is the value of the pixel being denoised, y_(i) is the valueof a particular pixel in the neighborhood of the pixel being denoised, Xrepresents denoising strength (i.e., a tunable factor to control howmuch noise is included), and a is the estimated noise level from thenoise model, e.g., the noise model described above. Thus, two pixels areconsidered similar if the absolute difference between them is less thana tunable, gain-dependent denoising strength parameter (based, at leastin part, on the amount of residual noise desired in the output image)times the estimated noise level for the pixel being denoised (asobtained from the noise model).

Most prior art noise models assume the same noise in Cb and Cr. However,in practice, this is often not actually the case. For example, AutoWhite Balance operations aren't unity, i.e., the same amount of gain isnot applied to each of the red, green, and blue channels. To illustrate,if the scene illuminant for a particular image happens to be a reddishlight (e.g., tungsten), then the gain of the blue channel will be muchlarger than the gain of the red channel. As a result, the noise in theCb channel would then be relatively larger than the noise in the Crchannel, due to the bigger gain being applied in the Cb channel. If aparticular noise reduction model assumes the noise to be equal in the Cband Cr channels, the noise reduction won't work well. If the model isdesigned just for Cb, then it will not adjust for noise in Cr. Thus, thenoise model may not be hand-tuned for a particular illuminant orchannel, because then the behavior will suffer for another illuminant orchannel. This behavior is the same for many pipeline operations, e.g.,gain, white balance, demosaicing, color correction, gamma, etc. That is,the noise behavior changes in some way with each of these operations, soit's difficult to characterize the noise based on any one operationperformed by the pipeline. Hence, it may be helpful to have both a lumaand a chroma noise model for each camera.

Turning now to FIG. 5, a block diagram of a multi-band luma denoisingscheme 500 is shown, according to one embodiment. (It is to beunderstood that the same scheme could also be applied to chroma datausing the aforementioned chroma noise models.) Once the signal has beendecomposed into individual sub-bands (e.g., in accordance with thefilter chain described with reference to FIG. 3 above), the multi-banddenoising scheme 500 may be applied to the image data. First, the Y_(H)band may be passed through sharpening filter 502. Sharpening filter 502may use a tuning parameter, k, in order to control the amount ofsharpness/fine grain amplitude desired for a particular band. Accordingto some embodiments, for bright scenes, there could be no attenuation ofthe sharpness filter, i.e., k=1. If more sharpness is desired, k couldbe greater than 1. For low light levels where pipeline artifacts becomemore visible, k may progressively become smaller, i.e., k<1.

Next, the lowest frequency band information, Y_(LLL), may be passedthrough the Y_(LLL) noise model 522 in order to perform pixel-basednoise reduction. The noise reduced data may then be up-sampled byup-sampler 524. The label Ŷ_(LLL) (“y-hat”) in FIG. 5 represents thenoise-reduced image information for the Y_(LLL) band. Then, a sharpeningfilter 526 may also be used on the noise-reduced data. This informationis then combined with the information from the next highest frequencysub-band, Y_(LLH), via addition junction 514. Then, the process of noisereducing, upscaling, and optionally sharpening is repeated at eachsub-band level before the resulting image information is combined withthe information from the next higher frequency band, until the highestfrequency band, Y_(H) (which, according to some embodiments is notnoise-reduced), is combined with, resulting in the reduced image lumaplane, Ŷ. A similar process may be performed independently on each ofthe chroma channels, using the respective chroma noise models for eachsub-band. As may now be understood, by sharpening the filtered sub-bandsindependently, the denoising scheme can enhance signal only—and notnoise (as is the case with sharpening masks that operate on unfilteredimage data).

As described above, according to some embodiments, noise reduction isonly performed on the low- to mid-frequency bands. The highest frequencyband is added back to the denoised frequency bands in order to get thefinal result. In this manner, the objectionable low-to-mid frequencynoise is removed while the high frequency noise, i.e., “blue noise,” isretained to convey a sharp impression—as well as mask certain pipelineartifacts. According to some embodiments, the cutoff point forfrequencies above which noise reduction is not to be performed could be“adaptive.” For low light conditions, where there is less high frequencyinformation, the cut-off frequency of the highest frequency band islowered to retain more noise, due to the fact that artifacts arerelatively large, and it is desirable to mask them and leave a texturedappearance. For bright light conditions, the cut-off frequency of thehighest frequency band is raised to retain only the very high frequency,pleasing “blue-noise”, due to the fact that the artifacts are smaller,and therefore not as important to be masked.

The noise models described herein may be further improved using channelcorrelation. For example, if two chroma pixels are adjudged to be‘similar’ to one another according to the ‘similarity score’ measurebeing used by the denoising algorithm, the algorithm may also attempt todetermine how similar the luma values for the pixels are, confirming the‘similarity’ determination of the algorithm based on chroma comparisononly if the luma values of the pixels are also within an acceptablesimilarity threshold. In this way, color edges within the image are lesslikely to degrade or “leak” across independent features in the image.

Adaptive Multi-Band Denoising Based on Local Noise Modeling

The center of a captured image is where the camera's lens has theability to capture the most signal. As pixels get farther and fartheraway from the center of the image, the lens response progressivelydecreases. At the corner of the image, the signal could be as low ashalf of the signal in the center. Thus, in order to achieve samebrightness across the image, the pixels on the sides and corners of theimage may be gained by the corresponding “lens shading factors,” butthis also increases noise in the corner areas of the images.

To boost dynamic range, a process known as “local tone mapping” (i.e.,LTM) boosts shadowy areas in the image to a greater extent thanhigh-light regions of the image, thus further amplifying noise in thoseregions. This can result in objectionable “crunchy” noise in the image.Typically, local tone mapping does not change much between individualadjacent pixels. Instead, it changes at a “regional” level within theimage—allowing the LTM curves for the image to be represented by afairly coarse LTM map

When multiple images are fused, in order to avoid “ghosting” artifacts,regions with moving objects, camera shake, rolling shutter, etc. in themmay be excluded from the fusion process, making the fusion strengthspatially-dependent, which, in turn, makes noise spatially-dependent. Inother words, the “no motion” areas in the image may use pixels from allthe frames, while regions that have motion or registration errors maynot use all the frames, thus resulting in a spatially-varying noisecharacteristic.

What is needed is an accurate and simple way to incorporate lens shadinggains, LTM, and fusion strength into a denoising algorithm, so that itlocally adapts to changing noise levels in the image. Thus, according toone embodiment, the inventors have developed a locally-adaptivemulti-band noise model that uses all the gains in the system, includingspatially-varying gains (e.g., lens shading, LTM, and fusion strength)as well as spatially-constant gains (e.g., analog and digital gains) toestimate an overall image gain level. This “overall gain” value may thenbe used by the noise model to estimate pixel noise.

Gain and noise levels have a highly non-linear relationship. Therefore,if spatial gains are simply used naively to drive denoising strengths,it will result in less denoising in low light and more denoising inbright light. By deriving an overall gain—and using the noise model topredict noise—a much more accurate noise estimate may be obtained.

In addition to spatially-dependent denoising/fusion, some embodimentswill also perform intensity-dependent fusion. This can serve to furthermitigate the “crunchy” look in shadow areas that results from local tonemapping. By relaxing the fusion constraint in low intensity regions,more denoising may be performed in the shadowy regions of the image,where there is less detail but more noise.

In bright regions of the image, the demosaicing block in the imageprocessing pipeline can cause aliasing in high contrast regions. Tomitigate this, some embodiments may use intensity-dependent de-moiréstrength within the denoising algorithm. That is, such embodiments mayincrease chroma denoising strength in bright regions, leading to lessvisible aliasing artifacts.

Turning now to FIG. 6, a block diagram of a multi-band locally-adaptivedenoising scheme 600 is shown, according to one embodiment. Multi-bandlocally-adaptive denoising scheme 600 is similar to the multi-banddenoising scheme 500 shown and described with reference to FIG. 5. Aprimary difference is that, for each sub-band, the pixel-based noisereduction (608/616/622) is done using a locally-adaptive noise model forthe respective sub-band. The locally-adaptive noise models may takelocal noise into account using one or more of the local noise modelingtechniques described above.

By splitting the luma and/or chroma signals into sub-bands and applyinglocally-adaptive noise reduction to all the bands below a givenfrequency and applying adaptive attenuation or amplification based onlight levels to the bands above the given frequency, and then summingthe bands to provide the full bandwidth signals, image enhancement isachieved. The YCbCr multi-band approach to denoising is able to operateindependently—and in an optimized fashion—on both luma and chromachannels. Noise reduction is done based on models developed for bothluma and chroma channels by measurements taken for multiple frequencybands, in multiple patches on the ColorChecker chart, and at multiplegain levels in order to develop a simple—yet robust—set of models thatmay be tuned off-line a single time for each camera and then applied toimages taken by such cameras in real-time without excessive processingrequirements and with satisfactory results across illuminant types andlighting conditions.

Referring now to FIG. 7, a simplified functional block diagram of anillustrative electronic device 700 is shown according to one embodiment.Electronic device 700 may include processor 705, display 710, userinterface 715, graphics hardware 720, device sensors 725 (e.g.,proximity sensor/ambient light sensor, accelerometer and/or gyroscope),microphone 730, audio codec(s) 735, speaker(s) 740, communicationscircuitry 745, digital image capture unit 750, video codec(s) 755,memory 760, storage 765, and communications bus 770. Electronic device700 may be, for example, a personal digital assistant (PDA), personalmusic player, mobile telephone, digital camera, or a notebook, laptop,or tablet computer system.

Processor 705 may be any suitable programmable control device capable ofexecuting instructions necessary to carry out or control the operationof the many functions performed by device 700 (e.g., such as thedenoising operations accordance with operations in any one or more ofthe Figures). Processor 705 may, for instance, drive display 710 andreceive user input from user interface 715 which can take a variety offorms, such as a button, keypad, dial, a click wheel, keyboard, displayscreen and/or a touch screen. Processor 705 may be a system-on-chip suchas those found in mobile devices and include a dedicated graphicsprocessing unit (GPU). Processor 705 may be based on reducedinstruction-set computer (RISC) or complex instruction-set computer(CISC) architectures or any other suitable architecture and may includeone or more processing cores. Graphics hardware 720 may be specialpurpose computational hardware for processing graphics and/or assistingprocessor 705 process graphics information. In one embodiment, graphicshardware 720 may include one or more programmable graphics processingunits (GPUs).

Sensor and camera circuitry 750 may capture still and video images thatmay be processed to generate images, at least in part, by video codec(s)755 and/or processor 705 and/or graphics hardware 720, and/or adedicated image processing unit incorporated within circuitry 750.Images so captured may be stored in memory 760 and/or storage 765.Memory 760 may include one or more different types of media used byprocessor 705, graphics hardware 720, and image capture circuitry 750 toperform device functions. For example, memory 760 may include memorycache, read-only memory (ROM), and/or random access memory (RAM).Storage 765 may store media (e.g., audio, image and video files),computer program instructions or software, preference information,device profile information, and any other suitable data. Storage 765 mayinclude one more non-transitory storage mediums including, for example,magnetic disks (fixed, floppy, and removable) and tape, optical mediasuch as CD-ROMs and digital video disks (DVDs), and semiconductor memorydevices such as Electrically Programmable Read-Only Memory (EPROM), andElectrically Erasable Programmable Read-Only Memory (EEPROM). Memory 760and storage 765 may be used to retain computer program instructions orcode organized into one or more modules and written in any desiredcomputer programming language. When executed by, for example, processor705, such computer program code may implement one or more of the methodsdescribed herein.

It should be emphasized that the previously described embodiments of thepresent invention, particularly any preferred embodiments, are merelypossible examples of implementations, set forth for a clearunderstanding of the principles of the invention. Many variations andmodifications may be made to the previously described embodiments of theinvention without departing substantially from the spirit and principlesof the invention. All such modifications and variations are intended tobe included herein within the scope of this disclosure and the presentinvention and protected by the following claims.

We claim:
 1. A non-transitory program storage device comprisinginstructions stored thereon to cause one or more processors to: receivesignals from an imager; process the signals received from the imager,wherein the processed signals comprise at least one luma signal and atleast one chroma signal; separate at least one of the processed lumasignals into a plurality of bands; separate at least one of theprocessed chroma signals into a plurality of bands; apply noisereduction to all bands of the at least one processed luma signal below afirst frequency, wherein a first luma noise model is used to estimate anoise level of each pixel of the at least one processed luma signal;apply noise reduction to all bands of the at least one processed chromasignal below a second frequency, wherein a first chroma noise model isused to estimate a noise level of each pixel of the at least oneprocessed luma signal; sharpen all bands of the at least one processedluma signal below the first frequency; sharpen all bands of the at leastone processed chroma signal below the second frequency; and sum all ofthe bands of the at least one processed luma signal and the at least oneprocessed chroma signal to produce an enhanced image signal.
 2. Thenon-transitory program storage device of claim 1, wherein theinstructions to apply noise reduction to all bands of the at least oneprocessed luma signal further comprise instructions to evaluate one ormore similarity thresholds for at least one pixel in the at least oneprocessed luma signal based, at least in part, on one or more of thefollowing: a band number of the respective pixel, an imager gain levelwhen the signals were captured, and a gain-dependent desired denoisingstrength.
 3. The non-transitory program storage device of claim 1,wherein the instructions to apply noise reduction to all bands of the atleast one processed chroma signal further comprise instructions toevaluate one or more similarity thresholds for at least one pixel in theat least one processed chroma signal based, at least in part, on one ormore of the following: a band number of the respective pixel, an imagergain level when the signals were captured, and a gain-dependent desireddenoising strength.
 4. The non-transitory program storage device ofclaim 1, wherein the instructions to sharpen all bands of the at leastone processed luma signal below the first frequency further compriseinstructions to sharpen all bands of the at least one processed lumasignal below the first frequency independently of one another.
 5. Thenon-transitory program storage device of claim 1, wherein one of theluma noise model and the chroma noise model comprises a linear model orpolynomial function model.
 6. The non-transitory program storage deviceof claim 1, wherein one of the first frequency and the second frequencyare tunable based, at least in part, on a brightness level of thereceived signals.
 7. The non-transitory program storage device of claim1, wherein the instructions to sharpen all bands of the at least oneprocessed chroma signal below the second frequency further compriseinstructions to sharpen all bands of the at least one processed chromasignal below the second frequency based, at least in part, on the imagergain level when the signals were captured.
 8. The non-transitory programstorage device of claim 1, wherein the first luma noise model and thefirst chroma noise model each depends on one or more of the following:signal value, analog and digital imager gains, white balance gains, anon-linear scaling function that scales full-band noise to each band,and a system-wide gamma estimate.
 9. A computer-implemented method forimage enhancement comprising: receiving signals from an imager;processing the signals received from the imager, wherein the processedsignals comprise at least one luma signal and at least one chromasignal; separating at least one of the processed luma signals into aplurality of bands; separate at least one of the processed chromasignals into a plurality of bands; applying noise reduction to all bandsof the at least one processed luma signal below a first frequency,wherein a first luma noise model is used to estimate a noise level ofeach pixel of the at least one processed luma signal; applying noisereduction to all bands of the at least one processed chroma signal belowa second frequency, wherein a first chroma noise model is used toestimate a noise level of each pixel of the at least one processed lumasignal; sharpening all bands of the at least one processed luma signalbelow the first frequency; sharpening all bands of the at least oneprocessed chroma signal below the second frequency; and summing all ofthe bands of the at least one processed luma signal and the at least oneprocessed chroma signal to produce an enhanced image signal.
 10. Thecomputer-implemented method of claim 9, wherein the act of applyingnoise reduction to all bands of the at least one processed luma signalfurther comprises evaluating one or more similarity thresholds for atleast one pixel in the at least one processed luma signal based, atleast in part, on one or more of the following: a band number of therespective pixel, an imager gain level when the signals were captured,and a gain-dependent desired denoising strength.
 11. Thecomputer-implemented method of claim 9, wherein the act of applyingnoise reduction to all bands of the at least one processed chroma signalfurther comprises evaluating one or more similarity thresholds for atleast one pixel in the at least one processed chroma signal based, atleast in part, on one or more of the following: a band number of therespective pixel, an imager gain level when the signals were captured,and a gain-dependent desired denoising strength.
 12. Thecomputer-implemented method of claim 9, wherein the act of sharpeningall bands of the at least one processed luma signal below the firstfrequency further comprises sharpening all bands of the at least oneprocessed luma signal below the first frequency independently of oneanother.
 13. The computer-implemented method of claim 9, wherein one ofthe luma noise model and the chroma noise model comprises a linear modelor polynomial function model.
 14. The computer-implemented method ofclaim 9, wherein one of the first frequency and the second frequency aretunable based, at least in part, on a brightness level of the receivedsignals.
 15. The computer-implemented method of claim 9, wherein the actof sharpening all bands of the at least one processed chroma signalbelow the second frequency further comprises sharpening all bands of theat least one processed chroma signal below the second frequency based,at least in part, on the imager gain level when the signals werecaptured.
 16. The computer-implemented method of claim 9, wherein thefirst luma noise model and the first chroma noise model each depends onone or more of the following: signal value, analog and digital imagergains, white balance gains, a non-linear scaling function that scalesfull-band noise to each band, and a system-wide gamma estimate.
 17. Asystem, comprising: an imager having a gain level; a memory having,stored therein, computer program code; and one or more processing unitsoperatively coupled to the memory and configured to execute instructionsin the computer program code that cause the one or more processing unitsto: receive signals from the imager; process the signals received fromthe imager, wherein the processed signals comprise at least one lumasignal and at least one chroma signal; separate at least one of theprocessed luma signals into a plurality of bands; separate at least oneof the processed chroma signals into a plurality of bands; apply noisereduction to all bands of the at least one processed luma signal below afirst frequency, wherein a first luma noise model is used to estimate anoise level of each pixel of the at least one processed luma signal;apply noise reduction to all bands of the at least one processed chromasignal below a second frequency, wherein a first chroma noise model isused to estimate a noise level of each pixel of the at least oneprocessed luma signal; sharpen all bands of the at least one processedluma signal below the first frequency; sharpen all bands of the at leastone processed chroma signal below the second frequency; and sum all ofthe bands of the at least one processed luma signal and the at least oneprocessed chroma signal to produce an enhanced image signal.
 18. Thesystem of claim 17, wherein the instructions to apply noise reduction toall bands of the at least one processed luma signal further compriseinstructions to evaluate one or more similarity thresholds for at leastone pixel in the at least one processed luma signal based, at least inpart, on one or more of the following: a band number of the respectivepixel, an imager gain level when the signals were captured, and again-dependent desired denoising strength.
 19. The system of claim 17,wherein the instructions to apply noise reduction to all bands of the atleast one processed chroma signal further comprise instructions toevaluate one or more similarity thresholds for at least one pixel in theat least one processed chroma signal based, at least in part, on one ormore of the following: a band number of the respective pixel, an imagergain level when the signals were captured, and a gain-dependent desireddenoising strength.
 20. The system of claim 17, wherein the first lumanoise model and the first chroma noise models each depends on one ormore of the following: signal value, analog and digital imager gains,white balance gains, a non-linear scaling function that scales full-bandnoise to each band, and a system-wide gamma estimate.